Skip to main content

Disaster Recovery Guide

Docker guide: https://docs.docker.com/engine/install/ubuntu/​

Restore Application Server​

Requirements & Prereqs​

  • Ubuntu 22.04 LTS or Ubuntu 24.04 LTS
  • Docker
  • nfs
  • nginx

Install NFS and nginx​

apt install nfs-common nginx

Mount NFS​

mkdir -p /mnt/backups/appbackup

fstab >>
192.168.0.31:/mnt/backups/appbackup    /mnt/backups/appbackup   nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0

mount -a

Extract backup​

tar -xvzpf /mnt/backups/appbackup/backup-file-name.tar.gz -C /

Setup directories​

mkdir -p /var/opt/zm/DataDumps/
mkdir -p /var/log/nginx/

Setup services​

  • Tailscale
  • node_exporter

Restart nginx​

systemctl restart nginx

Start SEIR services​

cd /opt/apps
docker compose up -d

cd /opt/apps/zm/unfepi
docker compose up -d

Reboot OS​

reboot

Verify​

Check the following for maintenance scripts and scheduled jobs

  • /root/scripts/
  • crontab

Restore Database Servers​

Requirements & Prereqs​

  • Ubuntu 22.04 LTS
  • MySQL 5.6 community edition
  • Percona xtrabackup
  • nfs

Install NFS​

apt install nfs-common

Mount NFS​

mkdir -p /mnt/backups/database

fstab >>
192.168.0.31:/mnt/backups/database    /mnt/backups/database   nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0

mount -a

Extract backup​

mkdir -p /tmp/mysql
tar -xvzpf /mnt/backups/database/xtrabackups/backup-file-name.tar.gz -C /tmp/mysql/

Install MySQL & Percona Xtrabackup​

  • Refer to the Deployment guides

Stop MySQL​

systemctl stop mysql

Setup directories​

mkdir -p /var/log/mysql
chown mysql:adm /var/log/mysql
chmod 750 /var/log/mysql

rm -rf /var/lib/mysql/*
mkdir -p /var/lib/mysql
chown -R mysql:mysql /var/lib/mysql

mkdir -p /tmp/xtrabackup_backupfiles
chmod 755 /tmp/xtrabackup_backupfiles

chown -R mysql:mysql /tmp/mysql/

Prepare Backup & Restore​

xtrabackup --prepare --target-dir=/tmp/mysql/backup-file-directory
xtrabackup --copy-back --target-dir=/tmp/mysql/backup-file-directory --datadir=/var/lib/mysql

Fix directory permissions​

chown -R mysql:mysql /var/lib/mysql

find /var/lib/mysql -type d -exec chmod 750 {} \;
find /var/lib/mysql-type f -exec chmod 640 {} \;

Start MySQL​

systemctl start mysql

Setup services​

  • Tailscale
  • node_exporter
  • PMM node setup

Verify & Test​

  • Check mysql logs
  • Test mysql connection

Replica setup​

  • Repeat the same steps for replica servers using the same backup file.
  • Use the replica cnf file provided in the deployment guide.